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ABSTRACT 

This paper aims to present an educational framework for outcomes 
based continuous quality improvement. Well defined program outcomes, 
program educational objectives and assessment process have been developed 
to ensure graduates’ outcomes achievement. Direct and indirect tools have 
been used for assessment process. Course evaluation surveys, alumni surveys, 
and employer surveys have been deployed for indirect outcome assessment. 
Exams, quizzes, assignments and projects, on the other hand, have been used 
for direct outcome assessment. In developed framework, the educational 
processes committees and facilities committees have been integrated to 
continuously evaluate and monitor the educational processes. Furthermore, 
program outcomes and course learning outcomes are proposed to be evaluated 
and continuously monitored by programs goals committee and continuous 
course improvement committee respectively. Forms and procedures have 
been developed to assess student outcomes. 


Outcomes based accreditation, 
program learning outcomes, 
course learning outcomes. 


1. INTRODUCTION 


International engineering alliance 
(IEA) gave a signatory status of Washington 
Accord (WA) to Pakistan Engineering Council 
in 2010. (Manual of accrediation Engineering 
Accrediation Board, 2014). Before 2002, the 
Washington Accord was based on convention- 
al education system widely known as input 
based education system. The major problem 
associated with conventional education sys- 
tem was non-provision of feedback; which 
was major hindrance in continuous quality 
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improvement (Patil and Codner, 2007). After 
the identification of drawbacks in conven- 
tional education system, Washington Accord 
adopted outcome based education system in 
2002 (Laguador and Dotong, 2014). After 
obtaining signatory status in 2002, Pakistan 
Education Commission faced the major chal- 
lenges of conversion of their education system 
from conventional to outcome based system. 
To overcome this issue, there was a need 
of continuous quality improvement framework 
for outcome based education implementa- 
tion. The framework was required to integrate 
course objective, curriculum mapping, direct/ 
indirect measure and continuous course qual- 
ity improvement (CCQI) as a single entity. 
The paper aims to propose a continuous 
quality improvement framework for engineer- 
ing education in Pakistan. The paper is struc- 
tured as follows. Literature review has been 
presented in Section 2. Section 3 describes 
the methodology adopted for the development 
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of framework. The proposed framework has 
been provided in Section 4. 

Section 5 presents the results and anal- 
ysis obtained by the application of proposed 
framework for a specific course. Finally Sec- 
tion 5 and 6 present conclusions and recom- 
mendations respectively. 


2. LITERATURE REVIEW 


Outcome based education is a newly de- 
veloped model for the improvement of quality 
education. (Rajaee et al., 2013). It focuses on 
teaching and learning based on desired stu- 
dent’s learning outcomes (Spady, 1994). To 
achieve these outcomes, assessment and per- 
formance standards have been designed and 
implemented (Spady, 1994). Literature review 
highlights the significance of outcome based 
education (Borsoto et al., 2014). Furthermore 
it serves as competitive advantage among dif- 
ferent countries (Spady, Marshall and Kit, 
1991). 

In past centuries, two major educa- 
tion systems have been proposed namely (1) 
problem based learning, and (11) project based 
learning. Problem based learning is a student 
centered learning approach. It encourages 
teachers to be facilitators instead of dissemi- 
nators. It 1s based on open ended problems 
which serve as initial stimulus (White, 2001). 
Project based learning, on the other hand, aims 
to engage students in authenticated problems 
investigation. It is based on project with the 
objective to provide learning (Blumenfeld et 
al., 1991). Both traditional educational sys- 
tems focus on contents with predefined cur- 
riculum and assessment system. Outcomes 
based education system, on the other hand, is 
relatively a new system which is based on de- 
fined framework for outcomes (Spady, 1994) 
1.e. it focuses on student’s outcomes and skills 
they must possess in future (Spady, Marshall 
and Kit, 1991). In this system, curriculum, 
assessments, instructional strategies and per- 
formance standards are developed and imple- 
mented to meet desired outcomes (Spady, 
1994), 

Outcome based education includes (1) 
program educational objectives (PEO’s), (11) 
program learning outcomes (PLO’s), (i) 
stakeholders involvement in defining PEO’s 
and PLO’s, a well curriculum development 
to achieve desired program educational ob- 
jectives and program learning outcomes, (iv) 
courses mapping to program educational ob- 
jectives and program learning outcomes for 


contribution and development of continuous 
improvement process (Rashid, 2012). 

Program educational objectives describe 
career and professional success that students 
attain after graduation (Manual of accredia- 
tion Engineering Accrediation Board, 2014). 
It 1s classified in to six categories namely (1) 
knowledge, (11) comprehension, (111) applica- 
tion, (iv) analysis, (v) synthesis and (v1) eval- 
uation (Student Outcomes and Program Edu- 
cational Objectives, 2015). Program learning 
outcomes have to be attained by students at 
the time of degree completion and must be 
specific, measureable, achievable, realistic, 
and time-based (Kapfer, 1971). 

Assessment is the third step which aims 
to defining clear and measureable outcomes, 
ensuring opportunities to achieve these out- 
comes, implementing an evaluation system 
and then use results from evaluation for stu- 
dent learning improvement (Bresciani, 2006). 
Many direct and indirect assessment tools 
have been proposed in literature. Direct as- 
sessment tools include course assessment, 
student satisfaction survey, cumulative GPA 
index for course, basic exams, senior design 
project, and program accreditation (Sekhar 
et al., 2008). Senior exit survey is an indi- 
rect program outcomes assessment tool (Oth- 
man et al., 2011). Teacher evaluation is a tool 
that can be used for the evaluation of teacher 
performance (Chalmers and Gardiner, 2015) 
by student feedback which helps to improve 
teaching skills. Teacher training programs 
are also designed on the basis of evaluation 
which aid to enhance teachers’ performance 
(Boerbooma et al., 2015). Rubrics is another 
indirect assessment tool that can be used for 
student’s skills and course assessment (AI- 
marshoud, 2011). Core indicators affecting 
academic institution are prioritized in three 
layers. Leadership, quality of faculty and in- 
frastructure facilities reside at first layer. Qual- 
ity of students, research, and learning environ- 
ment are in second layer followed by strategic 
planning, governance, assessment procedures 
and market force as the third layer (Jahanzaib 
and Akhtar, 2005). 

Review of literature presents that there 
exists a number of outcome based educa- 
tion models and continuous improvement 
frameworks. Bloom and his colleague Maser 
worked on behavioral objectives and devel- 
oped taxonomies of the objectives for cogni- 
tive and effective domain (King and Evans, 
1991). Davis adopted Carroll and Bloom’s 
work to achieve desired educational reform 
(Davis, 2003). The developed framework con- 
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sists of three phases. First phase of framework 
involves program and course objectives devel- 
opment; Second phase includes defining stu- 
dent learning outcomes based on Bloom’s tax- 
onomy of cognitive domain followed by third 
phase involving assessment workshops to fa- 
miliarize faculty and administration with as- 
sessment techniques (Mcgourty et al., 2002). 
Besterfield proposed a conceptual model for 
quality improvement in engineering education 
and developed program outcomes using affin- 
ity diagrams (Besterfield-sacre, Shuman, and 
Wolfe, 2002). Volkwein, Lattuca and TereN- 
zini developed a conceptual framework to 
examine impact of ABET’s EC200 impact on 
student’s learning outcomes (Volkwein, Lat- 
tuca, and TereNzini, 2004). Malaysian insti- 
tute of information and technology developed 
a computerized system to link, asses and mea- 
sure course learning outcomes and program 
learning outcomes (Mokhtar et al., 2014). Be- 
side these typical assessment models a new 
program outcomes assessment model focus- 
ing on performance criteria for each program 
outcome. The researcher discouraged the use 
of grades as performance indicators (Moham- 
mad and Zaharim, 2012). A continuous im- 
provement program based on outcomes based 
education comprised of student’s outcomes 
assessment tools have been developed and 
implemented by Sekhar, C. R. et all. (2008). 
An outcome driven hybrid academic model 
was proposed by Bouslama et al (2003). It 
uses grade point average along with outcomes 
driven students output. The model consists of 
learning outcomes, use of e-portfolios to re- 
cord student achievement, assessment and its 
reflection and closed feedback learning com- 
munities. 

From the detailed literature review, it 
can be seen that number of authors have pro- 
posed outcome based education implementa- 
tion models. However, it must be noted that 
systematic implementation of framework 1s 
required to fully achieve objectives of out- 
come based education. Engineering education 
institutions in Pakistan are bound to follow 
outcome based education in true spirit to get 
accreditation from Pakistan engineering com- 
mission. Therefore this research has proposed 
outcome based education continuous quality 
improvement framework. This will help the 
engineering institutions in Pakistan to apply 
outcome based education in true manners. 


Vol. 5, No.1, 2017. 


3. METHODOLOGY 


This research explores the continuous 
quality improvement framework for outcome 
based education implementation in engineer- 
ing institutions of Pakistan. The methodology 
consists of five phases. Literature review has 
been conducted in first phase to identify the 
limitations of previously developed frame- 
works. After identifying the gaps in phase I, a 
framework has been proposed in phase 2. The 
framework aligns the classroom practices and 
educational structure. A specific course was 
selected in phase 3 for the implementation of 
framework. Analysis of the course assessment 
data was conducted during phase 4. Finally, 
improvement methodology was suggested in 
phase 5 for continuous quality enhance. 


4. PROPOSED FRAMEWORK 


An outcomes assessment and continu- 
ous improvement framework based on above 
mentioned methods are proposed to ensure 
program educational achievement. The pro- 
posed framework has been presented in fig- 
ure |. 

The proposed framework consists of 
three modules namely (1) Outcome module, 
(11) management module and (111) assessment 
module. Outcome module defines the targeted 
outcomes and it consists of program educa- 
tional outcomes, program earning outcomes 
and course learning outcomes. Assessment 
module assesses the fulfillment of the targeted 
outcomes based on certain criteria. Manage- 
ment module consists of monitoring and eval- 
uation committees for continuous improve- 
ment. Three suggested committees are: (1) 
Continuous course improvement committee 
(CCI), (11) program goals committee (PGC) 
and (i111) education facilities and processes 
committee (EFPC). 

Significant feature of proposed frame- 
work is indicated by three layers of outcomes 
assessment cycles.. These cycles includes (1) 
course learning outcomes assessment, evalu- 
ation and continuous improvement cycle, (11) 
program learning outcomes assessment, eval- 
uation and continuous improvement cycle and 
(i111) program educational objectives assess- 
ment, evaluation and continuous improvement 
cycle. 

First cycle encloses course learning out- 
comes assessment and improvement cycle. 
Closing the loop takes sixteen weeks (one se- 
mester). The second cycle program outcomes 
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assessment cycle 1s a slow process and takes 
one year. Third cycle is program education- 
al objectives assessment and improvement 
which is also slow in nature. This cycle take 
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Figure 1. Outcomes Based Education Continuous Quality Improvement framework 


Outcomes Assessment methods used 
to evaluate outcomes according to developed 
framework are direct assessment and indirect 
assessments. Direct assessment includes out- 
comes assessment (e.g. course learning and 
program learning outcomes). Indirect assess- 
ment, on the other hand includes student’s 
course completion survey, exit survey, alumni 
survey and employer survey 

These assessment processes may be used 
to improve outcomes, course contents, cur- 
riculum and educational processes to achieve 
targeted outcomes. Teacher evaluation survey 
may be used to evaluate teacher performance. 

Following sections explains the imple- 
mentation of the proposed framework for 
course assessment and continuous improve- 
ment. The implementation starts with setting 
target outcomes (PEO, PLO and CLO), PLO/ 
CLO mapping, assessment and feedback for 
improving outcomes. All procedures been dis- 
cussed separately below. 


4.1. Setting target outcomes 


This step involves setting target pro- 
gram educational outcomes, program learn- 
ing outcomes and course learning. Program 
learning outcomes are statements which de- 
tail skills and abilities undergraduate students 
must achieve at time of degree completion. 
Program educational objectives and program 
learning outcomes have been developed as 
per Pakistan engineering council’s guidance 
and are presented in table 1. Each course in 
the entire industrial Engineering curriculum 
contributes to achieve some or many of the 
program learning outcomes. Correlation of 


current Industrial engineering courses with 
program learning outcomes has been present- 
ed in Annexure 1. For more detailed analysis, 
correlation of program learning outcomes of 
one of the selected course “Metal forming and 
cutting analysis” 1s shown in table 2. 
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Table 1. Program educational objectives and program learning outcomes 


Program Educational Objectives 


a 


To prodoce Indusinal Ensineers who assume challeneine manarenal and leadership roles m Manufac- 
turing and Service sectors both at National and Intemational Levels 

To provide Employers with eraduates who remain at forefront and are technically competent; have mtez- 
personal, decision makme and problem sohame shill: to enhance effectiveness 

‘To prodece professionals who recognize that engmeerme 15 a global sence profession that must be 
practiced ethicalh, with mteernty, honesty, and objectrethy 


Program Learning Outcomes 


i 


ira 


Enemeenne Knowledge: An ability to apply knowledge of mathematics, scremce and enzimeerme to 
solve complex problems. 

Problem Analysis: An ability to identify, formulate, solve and analyze engmeerme problems. 

Desien Development of Solutions: An ability to design a system, component, or process to meet desired 
needs within realistee constramts such as ecomomuc, environmental, social, political, ethecal, health and 
safety, manutacturability, and sustamabuility. 

Investigation: An ability to investigate complex engineering problems systematically such as literature 
Teview, desien and comduct experments, anahysis and interpretation of experiments to dere valid con- 
clusion 

Modem Tool Usage: An ability to create, select and apply appropiate techniques, skills, and modem en- 
Eineernme tools necessary for enemeernme practice 

The Engineer and Society: An ability to apply reasomme mformed by comtextual knowledge to assess 
societal health, safety, legal and cultural issues and the consequent responaibilities relevant to profes- 
clonal engineerme: practice and solutions to complex enemeenme problem: 

Environment and Sustamabilithy: Am ability: to understand the mupact of engineerme solutions ina 
global economucal, soctetal context, environmental and sustamable development 

Ei@ics: An ability to understand ethecal and professional responsibilities 

Individual and Team Work: An ability to work effectively as an mdividual or in a team 

Communication: An ability to communecate effectively 

Lifelonz Learmme: An ability to recoemee the need for immovation and technological development and to 
eneare in life-lonz leaning 

Project Manazement: An ability to apply concepts of industrial system and process design, erzonomucs, 
enpineenne manarement, product development'manufacturme and analysis tools te work m a team ef- 
fectrvely and to manage projects m mult disciplmary environments 


Table 2. Program learning outcomes for Metal forming and cutting analysis 


PLOs 
PLO l:a 
PLO2:b 
PLO3:c 


PLO4:e 


PLOs Statements 


an ability to apply knowledge of mathematics, science and engineering to solve complex problems 


an ability to identify, formulate, solve and analyze engineering problems 


an ability to design a system, component, or process to meet desired needs within realistic constraints 


such as economic, environmental, social, political, ethical, health and safety, manufacturability, and 


sustainability 


an ability to create, select and apply appropriate techniques, skills, and modern engineering tools 


necessary for engineering practice 


A set of measureable course learning experts took sixteen weeks to deliver course 
outcomes are required to drive the course. A contents assisting achieve course learning out- 
sample of course learning outcomes devel- comes. 
oped for the course “Metal forming and cut- 
ting analysis” are shown in table 3. CLO’s are 
developed on account of skills students must 
possess by the course completion. Course 
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Table 3. Course learning outcomes for 
Metal forming and cutting analysis 


CLO’s CLO’s statements 
Recalling the concepts of Manufacturing 
Processes for in-depth knowledge of pro- 
neice cesses for enhanced learning, Under- 
standing 
Identification of the known parameters in 
sie forming and cutting processes and making 
efficient use of boundary conditions AP- 
PLY 
Be able to apply related conditions (of 
parameters) and obtaining an appropriate 
—— solution within given constraints EVAL- 
UATE 
Reinforce evaluation capability for fur- 
ther synthesis of problems in metal form- 
CLO 4 
ing cutting analysis for creation of prod- 


ucts if any CREATE 


4.2. Correlation between Program 
learning outcome and course 
learning outcome 


Correlation assessment between course 
and program learning outcome allow program 
to know course contribution to outcomes but it 
do not allow assessment of specific outcomes 
achievement. For its assessment program 
learning outcomes are divided into number of 
measureable statements to identify student’s 
performance required to meet outcomes. 
These measureable statements are called per- 
formance criteria. For example PLO j “an 
ability to communicate effectively” would 
require criteria 1.e. (1) effective presentation 
(11) effective writing (111) professional commu- 
nication mastery. These three criteria require 
three types of assessments and measurement. 


Graduate must achieve all program outcomes 
by the end of degree completion which calls 
for a procedure to measure program learning 
outcomes by course completion. To measure 
PLO’s course learning outcomes are mapped 
to program learning outcomes by a linking 
matrix. In this way PLO’s are measured by ac- 
cumulated sum of CLO’s. For continuous 1m- 
provement course learning level monitoring is 
required to make sure contents delivered and 
assessed are in accordance with course com- 


plexity level or vice versa so, course learning 
outcomes (CLO’s) are mapped with Bloom’s 
taxonomy. In addition, course contents, teach- 
ing methods and assessment methods need to 
be documented and continuously monitored. 
Student satisfaction surveys conducted af- 
ter course completion assist teaching learn- 
ing methods, assessment methods and course 
contents continuous improvement where as 
a documentation procedure shown in table 4 
is developed which helps reviewing teaching 
learning methods, course contents and assess- 
ment methods in case of any CLO failure. 


Table 4. Curriculum mapping 


Learning Teaching Assessment 


CLO’s PLO’s Course contents 
Level methods method 
Under- Introduction to MFCA, joining processes Lecture, Quiz, Home 
stand and wear theory presentations, work 
Orthogonal Cutting Model in Machining, 
- Lectures, , 
Merchant Equation, Surface Finish and cut- Quiz, Home 
2 b Apply ji presentation, 
ting Tools Technology, Plastics extrusion, a work 
; discussion 
sheet metal cutting 
Forces calculations in metal cutting for 
Quiz, Home- 


evaluation, Forging analysis for pressure Presentations, 


Evaluate work, 


evaluation in billets, sheet metal cutting, discussion 


a . Exam 
grinding or machining process evaluation 


Create Mini Project/Capstone project Discussion Presentation 
Under a course offered by a degree pro- 
gram course learning outcomes are required to 
be directly and indirectly assessed and mea- 
sured. Table 5 shows procedure assisting out- 
comes measurement process. Assessments are 
the instruments used by instructor to assess 
student’s skills. One assessment method used 
for assessing a CLO is suggested but in pres- 
ent case instructor used more than one method 
to assess single CLO’s. Table 5 shows a ma- 
trix linking assessment questions to CLO’s to 
measure each CLO separately in addition with 


performance criteria for each assessment. 
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Table 5. Mapping assessments with CLO’s 


Assessment ; a 
question CLO Performance criteria 
techniques 
- an ability to apply knowledge of mathematics, science and engineering to 
HW 1 5 Qs 2 Raencness = Beare 
solve complex problems 
Home 7 = 
. HW 2 50s 2 an ability to identify, formulate, solve and analyze engineering problems 
Wor. 
‘ an ability to create, select and apply appropriate techniques, skills, and 
HW 3 5Qs 23 ee a ee ee 
modern engineering tools necessary for engineering practice 
; an ability to apply knowledge of mathematics, science and engineering to 
Quiz | 2 Qs Le 
solve complex problems 
Quiz Quiz2 2 Os ie an ability to identify, formulate, solve and analyze engineering problems 
. an ability to create, select and apply appropriate techniques, skills, and 
Quiz 3 2 Os 3 
modern engineering tools necessary for engineering practice 
Ql i Se 
Q2 Ze 
Mid- O3 an ability to identify, formulate, solve and analyze engineering problems 
Exam term O4 ‘ an ability to create, select and apply appropriate techniques, skills, and 
2 
exam modern engineering tools 
O45 3 necessary for engineerimg practice 
O6 
ability to create, select and apply appropriate techniques, skills, and mod- 
em engineering tools necessary for engineering practice 
Any Mini F ability to apply reasoning informed by contextual knowledge to assess 
other Project 7 societal, health, safety, legal and cultural issues and the consequent re- 


sponsibilities relevant to professional engineering practice and solutions 


to complex engineering problems 


percent marks to achieve desired program and 
course learning outcomes while 60 percent 
students must achieve course and program 
learning outcomes for a course to be success- 
ful. To measure particular course learning 
outcomes for a student «marks obtained by 
student are tabulated in excel form and then 
converted to percentage form. Then successful 
student’s percentage for particular CLO under 
the course is calculated to find achievement. 
Achievement indicators for outcomes are as 


4.3. Outcome assessment 


Course instructors must decide to setup 
instruments to assess course learning out- 
comes in their course. In present case Stu- 
dent’s skills and abilities are directly assessed 
by homework’s, quizzes, exams and projects. 

Indirect assessments are student’s per- 
ception based assessments on their outcomes 
achievement and understanding. Program 
learning outcomes (PLO’s)and course learn- 


ing outcomes (CLO’s) are indirectly assessed follows: ; 

by student’s satisfaction survey conducted © 90-100 % fully achieved 

after course completion. Student satisfaction - 80-90 4 excellently achieved 
survey was composed of two parts;1) Achiev- - 70-80 % achieved . 

ing of learning outcomes 2) course manage- ¢ 60-70 % satisfactoilry achieved 


ment. For this purpose likert scale (1-5) was * Below 60% failed 
used and students were asked to rate under- In case of course learning outcome 


standing of each CLO and PLO and course 
management which 1s described later. 

To evaluate student’s outcomes a thresh- 
old indicator is required to make sure achieve- 
ment of outcomes up to a certain level. This 
is to mention that student needs to obtain 60 


(CLO’s) correlation with single program learn- 
ing outcome (PLO’s) its data is directly used 
for PLO measurement but in multiple CLO’s 
correlation with single PLO case, weight must 
be defined for each CLO 1.e. Percentage con- 
tribution of CLO to achieve corresponding 
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PLO and final marks are composite sum of 
individual CLO’s at specific percentage and 
normalized to 100 percent. PLO achievement 
percentage is then measured by accumulat- 
ing relevant CLO’s data according to defined 
weights. Present case is an example of single 
CLO’s correlation with PLO. Mathematical 
model for PLO measurement is given below 


Step 1:V i determine percentage scores of 
students 
Step 2: V ¢t compute the average percent- 


age of scores to get ACO; average points 


Step 3: V it assign percentage weightages 
w to each j 

Step 4: Compute the sum of weightages by 
Yiewy » Vi 


Step 5: Compute the sum of Wie1WyXACO, Vj 

Step 6: Get the PO contribution by 

Vie Wij XACO; 
vies Wij 


, ¥ 


5. RESULTS AND ANALYSIS 


Figure 2 shows outcomes achievement 
for “Metal forming and cutting analysis” 
course. As course learning outcomes (CLO’s) 
mapping with program learning outcomes 
(PLO’s) shows one to one correspondence so 
the graph shows results for both PLO’s and 
CLO’s. It can be seen from figure 2 that suc- 
cessful student’s percentage is greater than 
60% for all PLO’s so the course 1s successful- 
ly completed with achievement of all CLO’s 
and PLO’s. 





CLO's Achivement 


| | | | 


Outcomes 


achieved 


not achieved 


Percentage achievement 











Figure 2. Course Learning Outcomes 
attainment 


5.1. Direct and indirect assessments 
result comparison 


Direct and indirect assessments need 


same basis of performance indicators to allow 
comparison. So Direct assessments percent- 
age marks were converted to indirect assess- 
ment scale. For this purpose students percent- 
age marks were divided by 20 to convert in to 
a linier scale 1.e. 1 - 5. Performance indicators 
for assessments are set as: For an outcome to 
be successful both direct and indirect assess- 
ment calculated value must be greater than 
three. Direct and indirect assessment results 
agree with indirect assessment results as given 
in table 6 and 7 which show students actual 
learning and understanding. It is evident to say 
that direct and indirect assessments are the in- 


struments to assess student’s outcomes. 


Table 6. CLO’s direct and indirect as- 
sessment comparison 


Direct assessment Indirect Assessment Remarks 
CLO Successful Scale=-5 Mean Standard 
students % deviation 

1 63 3.1 43 0.8 Successful 

2 75 3.7 43 0.6 Successful 

3 91 45 43 0.6 Successful 

- 3 3.7 41 0.6 Successful 


Table 7. PLO’s direct and indirect as- 
sessment comparison 


Direct assessment Indirect assessment Remarks 
PLO Successful Scale=5 Mean Standard 
students % deviation 
a 63 2.1 4.2 0.7 Successful 
b 75 3.7 43 0.7 Successful 
c 91 45 40 0.9 Successful 
e 75 | 4.0 0.9 Successful 


5.2. Measuring Course level by using 
Bloom’s taxonomy 


Engineering courses are being taught in 
an order of learning level and complexity 1.e. 
higher learning level course 1s offered in high- 
er semester. Continuous quality improvement 
calls a procedure for course learning level as- 
sessment for each course offered. To ensure the 
achievement of acceptable course level by an 
6instructor, selected course learning level was 
examined by mapping assessment questions to 
Blooms taxonomy. The frequencies of cogni- 
tive levels of assessment questions are given 
in table 8 for the specific course. It can be seen 
that 52% questions used for assessments were 
at understanding level, 8.6 % at application 
level and analyzing level, 21 % at evaluation 
level and only 8.6% at synthesis level. Maxi- 
mum marks percentage of each cognitive level 
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was evaluated. It can be established from table 
8 that the course “metal forming and cutting 
analysis” was taught at 5™ level of learning. 


Table 8. Assessing course learning level 


Questions Level Questions Max 


Percentage Percentage 
(Blooms) frequency marks 

Knowledge 0 0 0 0 
understanding 12 S217 5.75 19.16 
applying Z 8.6 1 a 
analyzing 2 8.6 8 42.6 
evaluation 5 21-f i275 40.8 
synthesis 2 8.6 8 26.6 
total 23 100 30 100 


5.3. Course evaluation 


Review of literature resulted in factors 
affecting student’s performance as shown in 
table 9 which are categorized into two parts: 
(1) outcomes and (2) course management. 
Former assesses course and program learning 
outcomes while later assess course contents, 
teaching learning methods and assessment 
methods. To assess course management Lik- 
ert scale 1-5 has been used. Table 9 details the 
means scores and standard deviation obtained 
in result of course evaluation survey analysis 
after course completion. 


Table 9. Course Evaluation survey 


Factors affecting student performance Mean SD 
Course * Clearly expressed 40 ay 
objective 
PLO x Attainment 3.8 0.8 


* Strengthened course un- 
derstanding 40 0.8 


contents x 


Course 
Helped achieving course 

objectives 

* Comprehensive 

Enhanced skills and abil- 4.1 0.9 


ities relevant to course 


Teaching ¥* 


learning 


methods 
* assessed CLO's 
* assessed Strength and 3.9 0.9 
Assess- weakness 
ments * helped Learning 
* Level was appropriate to 
given credit, 
CLO * Attainment assessment 4.02 0.8 
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5.4. Teacher Evaluation 


Review of literature resulted in mul- 
tiple teacher traits resulting positive students 
learning outcomes which can be stated as 
maintaining positive learning climate, class 
instructions, class management, and student 
involvement in class, teaching/learning and 
student differentiation. Table 10 details mean 
scores and standard deviation obtained in re- 
sult of teacher evaluation after course comple- 
tion. 


Table 10. Teacher Evaluation survey 


Teachers traits affecting students performance Mean SD 


* Student's respect 
; ‘ * Relaxed learning environment 
Learning Climate ae 46 06 
* Support for Class participation 
* Student collaboration 
* Declared lesson objective 
; ; * Well structured lesson 
Class instruction ; 42 08 
* Communication skills 
* Students comprehension 
*  Lecture’s start and finish time 
* Appropriate Students progress evalua- 
Class management ; = si 42 0.7 
tion techniques 
* Class room rules 
* Critical thinking encouragement by ask- 


Student involvement in ing questions 


i 41, “OF 

class * Student's interest in class 

* Students competition 

* Encourage students to think on solutions 

* Apply learning practically 
Teaching as learning * Problem solving strategies 41 08 

* Challenging assignments 

* Boost weak student's self confidence 
Student differentiation 3.3 1.3 


* Additional instructions time 


6. CONCLUSIONS 


Due to requirement imposed by Wash- 
ington accord agreement, engineering educa- 
tion in Pakistan is in the phase of transforma- 
tion from traditional education to object based 
education system. Program learning outcomes 
assessment and evaluation 1s mandatory now 
for all engineering programs in Pakistan. In 
order to ensure that the graduates of indus- 
trial engineering department have achieved 
program educational objectives, an outcomes 
based accreditation continuous improvement 
framework has been presented. The assess- 
ment of engineering courses with respect to 
learning outcomes and student’s expectations 
is an important activity regarding course 1m- 
provement. Therefore course learning out- 
comes were directly and indirectly assessed. 
It was found that the proposed method has 
the ability to gauge achievement of student’s 
outcomes. Strengths of the model include; (1) 
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clearly defined performance criteria which are 
evidence for specific program educational ob- 
jective, (11) performance criteria is linked with 
evidence, (111) use of performance indicators 
instead of grades. 

Continuous quality improvement can be 
implemented based on these strengths. Pro- 
posed method is a close loop model equipped 
with continuous quality improvement. 


7. SUGGESTIONS AND 
RECOMMENDATIONS 


aoe findings of this research include: 
The developed framework facilitates 
continuous quality improvement 
while developed forms and proce- 
dures assess students’ outcomes. 

¢ Student exit, alumni and employer 
surveys are tools to evaluate out- 
comes at program level while course 
evaluation survey is an instrument to 
evaluate outcomes at course level. 

¢ Student exit, teacher evaluation, and 
course evaluation survey are instru- 
ments to improve educational pro- 
cesses assisting attain educational 
objectives 

¢ Program goals committee evaluates 
and monitors outcomes at program 
level while educational processes 
and facilities committee evaluate 
and monitor program performance 
by conducting surveys. Continuous 
course quality improvement com- 
mittee 1s recommended to overcome 
course weaknesses. 

Procedure to improve course followed 

by Program outcomes assessment tools are 
described below 


7.1. Continuous course quality 
improvement 


The results analysis section showed 
both direct and indirect measures for the 
course selected. It 1s evident from direct and 
indirect measures that course 1s successful but 
some percentage of students failed in program 
learning outcome “a”, “b”, “c” and “e”. There 
is need for continuous course quality improve- 
ment monitored by CCQ/ committee to mini- 
mize failure percentage for same course next 
time. Given below methods are recommend- 
ed/ suggested to figure out failure reasons. 


7.1.1. Failure patterns 


To address this issue continuous course 
improvement committee (CCQI) committee 
systematically must use set of information re- 
lated to course and its prerequisite’s course: 

(1) outcomes correlation with the pre- 
requisite and present course 

(2) Students failure percentage in pre- 
requisite 

(3) switch from very low learning level 
(for pre-requisite) to very high learning level 
(present course) 

(4) pre-requisite Course contents 

(5) Level assessed in _ pre-requisite 
course exams. 


7.1.2. Issues related to student 
failure 


Survey may help identify issues related 
to student failure 1.e. poor course management 
or student’s personal problems. In case of poor 
course management actions must be taken to 
improve course. 


7.2. Program Learning Outcomes 
(PLO’s) and Program Educational 
Objectives (PEO’s) assessment tools 


Program learning outcomes and pro- 
gram educational objectives need to be con- 
tinuously assessed and monitored. Several 
tools have been developed to indirectly assess 
student’s outcomes at the time and after de- 
gree completion. Following section describe 
complete details of tools used for indirect as- 
sessment. 


7.2.1. Student Exit survey 


Review of literature resulted in number 
of factors affecting academic institutions per- 
formance and highlighted significance of stu- 
dent exit survey to evaluate institution perfor- 
mance and assess student’s outcomes. Factors 
affecting academic performance can be stated 
as: (1) Education quality (2) Facilities and (3) 
Curriculum. Student exit form has been devel- 
oped and categorized in to two sections: (1) 
Assessment of outcomes and (2) Academic 
performance evaluation. Table 10 show all 
factors enclosed in student exit survey in de- 
tail. 
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7.2.2. Alumni survey 


Alumni form also consists of two sec- 
tions: (1) PEO’s assessment and (2) alumni 
career. First section assesses outcomes while 
second section deals with graduate satisfaction 
with career, employment status, confidence in 
work and commitment to lifelong learning. 
Table 10 show all factors enclosed in Alumni 
survey in detail. 


7.2.3. Employer survey 


Employer form has been developed to 
collect employer’s views about alumni perfor- 
mance, employed at industries 1n addition with 
industry demands 1.e. which skills employers 
demand for an engineer to be employed at their 
organizations. Program educational objectives 
and program learning outcomes assessment in 
addition with modification to compete glob- 
ally is purpose of employer form as shown in 
table 11. 


Table 11. Student exit, Alumni and Em- 
ployer survey 


Factors 


Knowledge Slalls and * clearly declared 


attributes * Attainment assessment 
© Modify ‘delete/add 
Education quality * Instructions quality and knowledge 


* Professional practices development 

* Computational facilites 

* Laboratory facilites 

* Equipment needed for a modem education 
© Library holdings 


* Team projects and discussions 


Facilities 


Student Exit survey 


Cumiculum e Missing matenal 
* Unnecessary material 


e Career development 
General opinions * Important skills learned during graduation 
program educational © Meet newly admitted students 
Modify! delete / add 
* Current employment position 
Successful Career 
* Find job within | year after graduation 


objectives 


Satisfaction with career 


Graduate employment 

statuses 

* Jobis closely related to education 
Skills fulfill job demand 


* Engineering courses that need mmprove- 


Confidence in work 


Alumni Survey 


ments 

* Missing material 

Commitment to Ilife- 
long learnme 


* professional society member 
* Enrolled m education 

Academic and professional preparation 
* Industrial skills demand 

Modify delete’ add 


* Role in global Business 


Knowledge Slhills and 
attributes 


* Attamment assessment 
© Modify/delete/add 
Suggestions to improve 


Program educational 
objectives 


Employer survey 


General assessment 
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Annexure 1. Correlation Matrix 


Course Program Learning Outcomes 
Course Course Title a b c d 2 f E H 1. ] k 
Code Nene 
TE-101 Problem Solvime for swc Sct Mic Sic i Swcl Ne swc Swe Sw SWC 
Industnal Enzineers 
TE-102 WorkShop Practice sc = Swit 6S 1 Nc Swe Sc BAC swe Swe 
ME-19]1 9 Enemeernme Drawing and SC a sc Hc SC Hc Nc Nc ac bac STC 
Graphics 
ME-292 Mechanical Technology Sic SC Sc i Svc Nec Nic Mc Mc owe Nec 
MA-191 Calculus = SC a Swe | |=BI He Hc a Ne Sa Mc 
MA-192 Dnifferential Equations = SC SC Swit Sc aC Nic Mc Mic swe Ne 
3-192 Introduction to SC SC a Swe 86 He He a He i a 
Computime 
HU-112 0 [slamuc Studies / Ethics Nc Nc Mic Nc Mic Nc sc Mic Sac sc Mc 


TE-121 Probability and Statistics a SC sc Sc swic NC Nc Nc Nc SWC MC 


TE-122 Computer Aided Desen SC Sc sc Nic SC Mic Nc swic MC sc swe 
& Modeling 


-231 EnginessmzManazement SC SC SC Swe swt SWC SC SWCe swt sc NC 
ME-221 Emsimesrmz Mechanics SC SC SC Suc SC NC NC NC NC SUC Ne 
HU-101 Applied Physics sc sc sc sc swe NC NC NC NC Swe NC 


ME-293 Materials Enginesrmz sc sc sc swe sc swc NC NC NC SWC SWC 


MA-193 ApphedLimeasrAlepbra =SC SC SC SWC SC NC NC NC NC SWC NC 





TE-241 1 mz Eeonomics St SC a Sac 86S a Hc Sac NC Hc svc 

TE-245 Operations Research a a Sc Swe sc Svc NC NC NC Swe Swe 

TE-242 Manufacturing Process SC SC sc i SC Swit Nc Mic Mic swc SWC 

HU-113 Palostan Stodres Hc Nc Hc Nc Nc He SC Nec 7 Salt TC 

ME-294 Mechanics of Materials SC 1 a Sic Swit SWC Nc Nc Mc Sac Nc 

EE-301 Industrial Electronics i a SC Swe SC Nc Nc Nc Nc wo swe 

IE-311 Operations of SS SC a Sw |= BI SwWo MC Sa = MC wo SAC 
Moamufactiring Systems 

ME-311 Apphed Machme Design 5C SiC SC SC SC sac NC sac NC swe sac 
& FEM 2 1 

TE-312 8 §=©6TE-312 Metrology & ae SC SC swt 86st Nc Nec Nc Nc sc Svc 
Statistical Quality Control 
3] 

TE-313 [F-31535 Optmuzation SC sc SiC Svc 6S Swc NC NC Nc Sac Swe 
Techniques 2 0 

TE-314 JE-31]4 Work Stody & SC Sc SC swe Sc Swc NC SWC SWC SWC SWC 
Methods Enemeerme 

TE-321 Indusimal Simulaton 2 | SC = SC jh oe svwo ONC Sac NC 5c swe 

TE-322 [E-322 Human Factors a SC SC wo swe Sc Svc swe Suc «(SC Swit 


Engineering 2 | 

TE-325 [E-325 Management of Sut Sc Swit SWC SC Sac SWC SC Sc wo Swe 
Engineering Projects 3 0 

WMA-391) }4A-39] Numeral SC SC SC swe SC NC 8 860NC NC 8 060NC swe Nec 
Analysis 3 0 

TE-324 Planing & Schedulmeim SC 2 SC swe 6SC swe Ne Sal Mc swe SWC 
Manufacturing 2 0) 





TF-325 Industrial Automaton SC sc SC Swco | 6SC sc Nec Swe Ne 5c sWec 
and Robotics 

TE-411 Design of Experments 3 SC Sc SiC SC SC SC Nc SWC NC Sc Swit 
] 

TE-412 IE-412 Industnal SC 5 SC Swe 8 6ScC 5c Ne SVC Ne Swo SWC 
Facilities Design 2 1 

IF-33 OJE-XX Elecine I Swol SWC SRC 6ST UCSC SVC SWC 5C sc Sac 6S 
(HRM) 

IE-3XX | OJE-XXYX Electrve I SC a SC Sai 6S 5c Nc Hc Hc So SWeo 
(Metal fommine an cuttme 
analysis 

TE-491 JE-491] Project Phase | SC SC 5c SC sc Sc sc sc Sc SC sc 

TE-SOOX Elective! (Organizational SWC SWC SWC SWC SWt Sut SC SC swt SWC Swe 
behavior 

TF-X «= JE-X XX Electrve I SC SC SC Swe 6st 5c Ne oe Ne | Sac 
(Automation and ctl) 

IE-X3S | OJE-XXY Electrve I SC ov SC swt 6S sc Nc Swo Ne 5c Sa 


(CIM) 
TESOEX [E-3OCX Elective 


TE-492 = TE-492 Project Phase I SC SC SC SC SC 1 SC = ac SC SC 


SC: Strongly correlated, SWC: Somewhat correlated, NC: Not correlated 
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